execution, cl/phase1: stop discarding a payload that is already built - #23289
execution, cl/phase1: stop discarding a payload that is already built#23289lystopad wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR is a follow-up in the Caplin ↔ execution-module integration that prevents a fully-built payload from being discarded when BlockBuilder.Stop is called with a canceled/expired caller context, and it unifies the “busy” sentinel error identity across call sites.
Changes:
- Make
BlockBuilder.Stopprefer a finished payload over a canceled/expired caller context to avoid nondeterministically dropping ready blocks. - Move the execution-module “busy” sentinel to
execution/execmoduleand update callers/tests to use the shared identity (errors.Isworks across packages). - Improve retry behavior and cancellation reporting: use
common.Sleep(ctx, d), preserve the last contention error when timing out/canceling, and tighten test delays to avoid hour-long hangs.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| execution/execmodule/set_head.go | Wraps semaphore acquisition failures with the shared execmodule.ErrBusy sentinel. |
| execution/execmodule/interface.go | Introduces execmodule.ErrBusy sentinel next to the Busy field it represents. |
| execution/execmodule/chainreader/chain_reader.go | Switches busy returns from the old chainreader sentinel to execmodule.ErrBusy. |
| execution/execmodule/block_building.go | Avoids error-level logging for caller cancellation/deadline in GetAssembledBlock. |
| execution/builder/block_builder.go | Updates Stop to deterministically prefer an already-finished payload over ctx.Done(). |
| execution/builder/block_builder_test.go | Adds a regression test to ensure Stop returns the finished payload even with a canceled caller context. |
| cl/phase1/execution_client/execution_client_direct.go | Updates assemble retry logic to wait via common.Sleep and key off execmodule.ErrBusy. |
| cl/phase1/execution_client/execution_client_direct_test.go | Updates retry tests to use execmodule.ErrBusy and shorter delays. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if err := e.semaphore.Acquire(acquireCtx, 1); err != nil { | ||
| return fmt.Errorf("execution module is busy: %w", err) | ||
| return fmt.Errorf("%w: %w", ErrBusy, err) | ||
| } |
There was a problem hiding this comment.
Good catch. Acquire fails for two different reasons and I flattened them: the local 5s timeout, which does mean the module is occupied, and the caller's own context going away, which says nothing about it. Marking the second as busy invites a retry with nothing to wait for.
Only the timeout reports ErrBusy now. The conflation predates this PR, but it was harmless while the error was untyped — making it a sentinel is what turned it into something callers can branch on, so it belongs here. Covered by TestSetHeadReportsBusyOnlyWhenTheModuleIsOccupied, which fails against the previous wrapping.
| // A caller that gave up says nothing about the builder, which keeps running. | ||
| if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { | ||
| return AssembledBlockResult{}, err | ||
| } |
There was a problem hiding this comment.
You are right — Stop sets interrupt before it waits, so the builder is stopped either way and the comment described something that does not happen. Reworded to the actual point: the caller gave up waiting, and nothing about the build itself went wrong.
3de271c to
17f9e3c
Compare
yperbasis
left a comment
There was a problem hiding this comment.
Requesting changes for one remaining race in BlockBuilder.Stop. The current priority probe does not cover completion between the outer and inner selects, so the payload-loss behavior can still occur. Please address the inline finding.
| default: | ||
| select { | ||
| case <-ctx.Done(): | ||
| return nil, ctx.Err() |
There was a problem hiding this comment.
b.done is only prioritized by the probe above. If it closes after that probe but before this inner select chooses, both ctx.Done() and b.done are ready and Go may still select this branch, returning cancellation while the completed payload is available. This leaves the proposal-loss race in place, only with a smaller window. Please non-blockingly recheck b.done here before returning; a regression test can force completion between the probe and selection.
There was a problem hiding this comment.
You are right — the probe narrowed the window rather than closing it. If the payload lands between the probe and the select's choice, both channels are ready again and it is back to a coin flip.
The cancellation branch now re-checks before returning, so a payload that has landed is never traded for it. The remaining window is only where the payload genuinely had not landed when the decision was made, which is a real race rather than a scheduling bias.
On forcing it: my first attempt at a regression test did not reproduce the bug at all — the payload landed before Stop even probed, so the outer check caught it and the test passed against the broken version too. What works is closing the completion channel from the context's own Done method, which the select reads while setting itself up. That puts the completion exactly between the probe and the choice on every iteration. Removing the re-check makes it fail immediately; with the re-check, 300 iterations pass every run.
17f9e3c to
a6cc442
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (1)
execution/execmodule/set_head.go:59
- This still classifies the failure by reading the parent context after
Acquirereturns. If the five-second timeout releasesAcquireand the caller is canceled before this check, the local timeout is incorrectly returned withoutErrBusy. Latch the timeout source withWithTimeoutCauseand inspect the derived context's cause instead.
if ctx.Err() != nil {
|
Also took the suppressed comment about The wait now carries its own cause via |
yperbasis
left a comment
There was a problem hiding this comment.
Nits only, none blocking. The red mainnet-rpc-integ-tests check is an infra flake (HTTP 429 while downloading actions/checkout), unrelated to this PR.
execution/execmodule/block_building.go:134: the gate classifies by error shape, butBuildruns on the node lifecycle ctx, not the caller's. A genuine build failure that wraps a context error — e.g. Shutter's parent-block wait timeout, or node shutdown surfacing throughBeginTemporalRo— skips the ERROR log while the caller is still alive, and the comment above is then wrong. A precise alternative: letStopmark its ownctx.Done()branch with a typed sentinel and gate on that.execution/execmodule/set_head.go:64: this is now the only unwrapped error return inSetHead;fmt.Errorf("set head: %w", err)would keep the operation visible in the error. On line 62,%w: %walso makes the busy error satisfyerrors.Is(err, context.DeadlineExceeded); if that matchability is unwanted,%vfor the inner error avoids it with the same message.execution/execmodule/set_head_internal_test.go:42: the new test pins the stdlib cause-latch but never callsSetHead, so deleting theErrBusywrap on set_head.go line 62 still leaves the package green. An unexported acquire-timeout field onExecModule(the tests already build the struct literally) would let the busy arm run throughSetHeadin milliseconds.execution/execmodule/interface.go:101: "Busy is true when the builder has not finished yet" describes the wrong condition — it is true when the module semaphore was contended; otherwiseGetAssembledBlockblocks inStopuntil the builder finishes. It now also contradicts theErrBusydoc a few lines above.cl/phase1/execution_client/execution_client_direct_test.go:71: with a 1s delay the test also passes ifcommon.Sleepis replaced by a plaintime.Sleep— the loop-top check then returns an error of the same shape after the sleep, so the "abort during the backoff wait" property is no longer pinned. Asserting the test finishes well under the 1s delay restores it.cl/phase1/execution_client/execution_client_direct.go:167: theerr != nilarm duplicates the Sleep-site wrap and looks unreachable —assemblenever blocks on ctx, andcommon.Sleepalready returns the wrap on cancellation; only a cancellation landing in the gap after Sleep returned nil hits it. A plain ctx check hoisted above the loop keeps one wrap site.execution/builder/block_builder.go:96: "The second check matters as much as the first" overstates the outer probe —donenever reopens, so the inner re-check already covers it, and both tests stay green with the probe deleted. Worth rewording so nobody later keeps the probe and drops the re-check, which is the one that prevents the bug. Line 94: "however close together the two arrive" → "even when both are ready at the same time" is easier to parse.- Optional, on-theme:
cl/beacon/handler/block_production.go:259logs ERROR for every poll failure, includingexecmodule.ErrBusy— the routine contention the poll loop exists to wait out. The now-exported sentinel makes a one-line demotion to Debug possible.
domiwei
left a comment
There was a problem hiding this comment.
I found a few remaining issues on a6d145b:
-
Cancellation is lost on the final busy attempt (
cl/phase1/execution_client/execution_client_direct.go:172-186). Ifassemblecancels the context and returnsexecmodule.ErrBusyon the last attempt, theattempt+1 == attemptsbranch breaks and returns onlyErrBusy. This makes cancellation classification depend on the retry position. A deterministic reproducer isattempts=1, with the callback callingcancel()and returningErrBusy; the returned error does not matchcontext.Canceled. Please recheckctx.Err()before the final busy return and preserve both causes, as the earlier-attempt paths do. -
Removing
chainreader.ErrExecutionBusybreaks source compatibility. It was exported on the base branch, so downstream code referring to it no longer compiles. The canonical identity can still move toexecmodule.ErrBusywhile retaining a compatibility alias such asvar ErrExecutionBusy = execmodule.ErrBusyinchainreader. -
SetHeadcan mistake a caller-supplied cancellation cause for local contention (execution/execmodule/set_head.go:54-63).ErrBusyis used both as the exported classification and as the derived context's private timeout marker. Withctx, cancel := context.WithCancelCause(parent); cancel(ErrBusy),context.Cause(acquireCtx)is alsoErrBusy, soSetHeadreturns an error matchingErrBusyeven though only the caller went away. I reproduced this against the PR head. Please use a private sentinel for the local timeout and map only that sentinel to the publicErrBusy. -
The positive
SetHeadtest does not exerciseSetHead(execution/execmodule/set_head_internal_test.go:41-55). It independently constructs aWithTimeoutCause, calls the semaphore, and checkscontext.Cause; it would still pass ifSetHeadstopped returningErrBusy. Please cover the actual acquisition-failure path and assert the public returned error identity.
The updated BlockBuilder.Stop handoff looks correct now: the post-cancellation completion probe closes the remaining select window. Focused existing tests and CI are green; the two deterministic reproducers above still fail on the current head.
a6d145b to
d228f18
Compare
|
@domiwei — all four fixed in Cancellation lost on the final busy attempt. Right: with no attempts left there is no wait to notice it, so how the caller got classified depended on which retry it died on. The last attempt is checked directly now and keeps both causes.
The positive test not exercising Source compatibility. Taken — Also took @yperbasis's non-blocking notes on this branch: the |
d228f18 to
182592b
Compare
yperbasis
left a comment
There was a problem hiding this comment.
Requesting changes on the two blocking items; the rest are non-blocking.
Blocking
execution/execmodule/block_building.go:134: the gate classifies by error shape, butBuildruns on the node lifecycle ctx, not the caller's. A genuine build failure that wraps a context error (Shutter's parent-block wait timeout, shutdown duringBeginTemporalRo) is treated as "the caller gave up": the ERROR log is skipped and the comment above it is wrong. LetStoptag its ownctx.Done()give-up branch with a typed sentinel and gate on that instead. Flagged in the previous review as well as by a second reviewer.execution/execmodule/interface.go:160: theBusyfield doc fix did not reach theGetAssembledBlockmethod doc, which still says "The result is Busy when the builder has not finished" — the file now contradicts itself. Same one-line fix as the field doc.
Test coverage (non-blocking)
execution/builder/block_builder.go:103: the cancellation branch (return nil, ctx.Err()) has no test — deleting it leaves the suite green, and if it is ever lost, a caller cancellation turns intoUnknownPayloadErrat the engine API instead of a context error. A cancelled ctx plus an unfinished build pins it.execution/execmodule/set_head.go:71: the three SetHead tests pin outcomes but not the mechanism — classifying onerrors.Is(err, context.DeadlineExceeded)instead of the cause passes all three while misreporting a caller's shorter deadline as busy. A fourth case (acquireTimeout of an hour, caller timeout of 1ms → not busy) kills that mutant.cl/phase1/execution_client/execution_client_direct_test.go:80: 500ms against a 1s delay leaves little margin on a loaded CI runner, and a plaintime.Sleep(200ms)still passes.delay = time.Minutewith a ~10s bound keeps the property with margin in both directions.
Minor
cl/phase1/execution_client/execution_client_direct.go:192: the retry path keepsErrBusymatchable in cancelled-caller errors whileSetHeadstrips it — both defensible, but one doc line onErrBusystating the convention would keep a futureerrors.Is(err, ErrBusy)retry trigger from firing for a dead caller.cl/phase1/execution_client/execution_client_direct.go:163:ranOutduplicateserrors.Join(precedent:db/integrity/torrent_verify.go:54); worth keeping only if the single-line message shape is load-bearing.
Follow-up to #23273, which made this reachable: BlockBuilder.Stop selected on the caller's context and on the finished payload at once, so when both were ready Go chose between them at random and about half the time returned a cancellation while holding a complete block. The caller had passed context.Background() before, so the race could not fire; now a validator client that times out can lose a proposal that was ready. The finished payload wins. A caller that gave up is also not a build failure, and was being reported as one. It now returns without an error-level record. The busy sentinel moves next to the Busy field it reports, so set_head.go's identically worded error shares its identity instead of only its wording. The hand-rolled cancellable sleep becomes common.Sleep, and the contention that caused a wait is kept in the error rather than replaced by the bare context error.
A caller that goes away while waiting for the semaphore is not a busy module, and saying so would invite a retry with nothing to wait for. Only the local timeout reports ErrBusy. The conflation predates this change but was harmless while the error was untyped. Correct a comment that claimed a cancelled caller leaves the builder running: Stop interrupts it either way, and the point is only that this is not a build failure.
Probing before the select only narrowed the race: the payload can land while the select is choosing, and both channels are then ready again. Re-check after cancellation is chosen, so a payload that has landed is never traded for it. The regression test closes the completion channel from the context's own Done call, which is read as the select sets itself up, so the interleaving happens on every attempt rather than being a few nanoseconds wide.
Reading the caller's context after Acquire returns misreads a caller that went away just after the local timeout fired, which is the case that says the module was occupied. The wait now carries its own cause, recorded at the moment it ran out.
The assemble retry lost the caller's cancellation when it landed on the final attempt: there was no wait left to notice it, so how the caller was classified depended on which retry it died on. The last attempt is now checked directly, keeping both causes. SetHead classified on the exported sentinel, which a caller can supply as its own cancellation cause and have contention reported for its own timeout. The marker is private now, and the wait's length is a field so the busy path can be reached through SetHead itself rather than reconstructed beside it. The busy sentinel keeps an alias where it used to be defined, so callers that referred to it still compile. Also: the Busy field described the builder still working rather than the module being occupied; the retry test could not tell a cancellable wait from a plain sleep; and the comment on the payload priority check credited the shortcut rather than the check that does the work.
The wait can only have failed on its own deadline in that branch, which is what ErrBusy already says, so the inner error added the deadline back as unmatchable text.
… be guessed A build can fail with a context error of its own - a transaction provider timing out, a shutdown reaching the read view - so the error's shape does not say whether the caller gave up or the build failed. Reading it that way skipped the record for a real failure. Stop tags its own give-up branch instead, and that is what the caller checks. The GetAssembledBlock method doc described Busy as the builder not having finished, which the field doc had already been corrected away from. The file no longer contradicts itself. Three tests that pinned outcomes without pinning the mechanism now do both: Stop giving up rather than failing, a caller whose own deadline is shorter than the module's wait, and a backoff that aborts rather than sleeping through cancellation. Each fails against the substitution it is there to rule out. ErrBusy documents the convention it is used under, since it stays matchable inside a cancelled caller's error but must not on its own trigger a retry for a caller that has gone.
182592b to
517b7bf
Compare
|
All of it in Classifying by error shape. You and @domiwei both raised this and I answered it twice without fixing it — I argued the returned error was enough, then narrowed it to
The The three coverage gaps. All three were fair, and all three describe tests of mine that pinned an outcome without pinning the mechanism. I checked each against the substitution you named rather than assuming:
Minor.
|
Follow-up to #23273, addressing @yperbasis's post-merge review. Part of the series splitting #23105.
A finished payload could be thrown away
BlockBuilder.Stopselected on the caller's context and on the finished payload at once. When both are ready Go picks between the cases at random, so roughly half the time it returnedcontext canceledwhile holding a complete block.This was unreachable before #23273 because the caller passed
context.Background(). Now that a real context reaches it, a validator client that times out — or any request cancellation during the proposal window — can lose a payload that was ready to publish. The finished payload now wins.TestBlockBuilderStopPrefersAFinishedPayloadOverAnExpiredCallerdrives 50 iterations with both ready; it fails on every run against the previous ordering.Cancellation is not a build failure
GetAssembledBlockloggedERROR "Failed to build PoS block" err=context canceledwhen the caller simply gave up — routine on a healthy node once a cancellable context reaches it. It now returns the error without an error-level record. Classification comes from the returned error rather than the ambient context, which cannot drift betweenStopreturning and the check.One identity for the busy signal
The sentinel moves to
execution/execmodule, next to theBusyfield it reports, soset_head.go's identically worded error shares its identity rather than only its wording.errors.Isnow works across both.Smaller points from the same review
common.Sleep(ctx, delay), and the drain branch it had was dead;context.Canceled— that cause is the diagnostic cl/phase1, execution: give the execution module a typed busy signal and the caller's context #23273 introduced;time.Hourwaits in the retry tests aretime.Second, so a regression fails as an assertion rather than as a package timeout.Left for later, as suggested: the server-side
Acquire(ctx)question,ErrUnknownPayload, and the LittleEndian/BigEndian payload-id mismatch on theisLocalengine path.